Knowledge-Based Visual Question Answering Using Multi-Modal Semantic Graph

The field of visual question answering (VQA) has seen a growing trend integrating external knowledge sources to improve performance. However, owing the potential incompleteness and inherent mismatch between different forms data, current knowledge-based (KBVQA) techniques are still confronted with challenge effectively utilizing multiple heterogeneous data. To address this issue, novel approach centered on multi-modal semantic graph (MSG) is proposed. MSG serves as mechanism for unifying representation data diverse types knowledge. Additionally, reasoning model (MSG-KRM) introduced perform deep fusion image–text information sources. development involves extracting keywords from image object detection information, text, texts, which then represented symbol nodes. Three graphs constructed based graph, including vision, question, non-symbol nodes added connect these three independent marked respective node edge types. During inference stage, embedded into feature through embedding methods, type-aware attention module employed reasoning. final answer prediction blend output pre-trained model, pooling results, characteristics non-symbolic experimental results OK-VQA dataset show that MSG-KRM superior existing methods in terms overall accuracy score, achieving score 43.58, improved most subclass questions, proving effectiveness proposed method.